Measuring the perception and metacognition of time.

The ability of humans to identify and reproduce short time intervals (in the region of a second) may be affected by many factors ranging from the gender and personality of the individual observer, through the attentional state, to the precise spatiotemporal structure of the stimulus. The relative roles of these very different factors are a challenge to describe and define; several methodological approaches have been used to achieve this to varying degrees of success. Here we describe and model the results of a paradigm affording not only a first-order measurement of the perceived duration of an interval but also a second-order metacognitive judgement of perceived time. This approach, we argue, expands the form of the data generally collected in duration-judgements and allows more detailed comparison of psychophysical behavior to the underlying theory. We also describe a hierarchical Bayesian measurement model that performs a quantitative analysis of the trial-by-trial data calculating the variability of the temporal estimates and the metacognitive judgments allowing direct comparison between an actual and an ideal observer. We fit the model to data collected for judgements of 750 ms (bisecting 1500 ms) and 1500 ms (bisecting 3000 ms) intervals across three stimulus modalities (visual, audio, and audiovisual). This enhanced form of data on a given interval judgement and the ability to track its progression on a trial-by-trial basis offers a way of looking at the different roles that subject-based, task-based and stimulus-based factors have on the perception of time.

Event Probabilities Have a Different Impact on Early and Late Electroencephalographic Measures Regarded as Metrics of Prediction.

The oddball protocol has been used to study the neural and perceptual consequences of implicit predictions in the human brain. The protocol involves presenting a sequence of identical repeated events that are eventually broken by a novel "oddball" presentation. Oddball presentations have been linked to increased neural responding and to an exaggeration of perceived duration relative to repeated events. Because the number of repeated events in such protocols is circumscribed, as more repeats are encountered, the conditional probability of a further repeat decreases-whereas the conditional probability of an oddball increases. These facts have not been appreciated in many analyses of oddballs; repeats and oddballs have rather been treated as binary event categories. Here, we show that the human brain is sensitive to conditional event probabilities in an active, visual oddball paradigm. P300 responses (a relatively late component of visually evoked potentials measured with EEG) tended to be greater for less likely oddballs and repeats. By contrast, P1 responses (an earlier component) increased for repeats as a goal-relevant target presentation neared, but this effect occurred even when repeat probabilities were held constant, and oddball P1 responses were invariant. We also found that later, more likely oddballs seemed to last longer, and this effect was largely independent of the number of preceding repeats. These findings speak against a repetition suppression account of the temporal oddball effect. Overall, our data highlight an impact of event probability on later, rather than earlier, electroencephalographic measures previously related to predictive processes-and the importance of considering conditional probabilities in sequential presentation paradigms.

Predictive extrapolation effects can have a greater impact on visual decisions, while visual adaptation has a greater impact on conscious visual experience.

Human vision is shaped by historic and by predictive processes. The lingering impact of visual adaptation, for instance, can act to exaggerate differences between past and present inputs, whereas predictive processes can promote extrapolation effects that allow us to anticipate the near future. It is unclear to what extent either of these effects manifest in changes to conscious visual experience. It is also unclear how these influences combine, when acting in concert or opposition. We had people make decisions about the sizes of inputs, and report on levels of decisional confidence. Tests were either selectively subject to size adaptation, to an extrapolation effect, or to both of these effects. When these two effects were placed in opposition, extrapolation had a greater impact on decision making. However, our data suggest the influence of extrapolation is primarily decisional, whereas size adaptation more fully manifests in changes to conscious visual awareness.

On why we lack confidence in some signal-detection-based analyses of confidence.

Signal-detection theory (SDT) is one of the most popular frameworks for analyzing data from studies of human behavior - including investigations of confidence. SDT-based analyses of confidence deliver both standard estimates of sensitivity (d'), and a second estimate informed by high-confidence decisions - meta d'. The extent to which meta d' estimates fall short of d' estimates is regarded as a measure of metacognitive inefficiency, quantifying the contamination of confidence by additional noise. These analyses rely on a key but questionable assumption - that repeated exposures to an input will evoke a normally-shaped distribution of perceptual experiences (the normality assumption). Here we show, via analyses inspired by an experiment and modelling, that when distributions of experience do not conform with the normality assumption, meta d' can be systematically underestimated relative to d'. Our data highlight that SDT-based analyses of confidence do not provide a ground truth measure of human metacognitive inefficiency. We explain why deviance from the normality assumption is especially a problem for some popular SDT-based analyses of confidence, in contrast to other analyses inspired by the SDT framework, which are more robust to violations of the normality assumption.

The spatial properties of adaptation-induced distance compression.

Exposure to a dynamic texture reduces the perceived separation between objects, altering the mapping between physical relations in the environment and their neural representations. Here we investigated the spatial tuning and spatial frame of reference of this aftereffect to understand the stage(s) of processing where adaptation-induced changes occur. In Experiment 1, we measured apparent separation at different positions relative to the adapted area, revealing a strong but tightly tuned compression effect. We next tested the spatial frame of reference of the effect, either by introducing a gaze shift between adaptation and test phase (Experiment 2) or by decoupling the spatial selectivity of adaptation in retinotopic and world-centered coordinates (Experiment 3). Results across the two experiments indicated that both retinotopic and world-centered adaptation effects can occur independently. Spatial attention to the location of the adaptor alone could not account for the world-centered transfer we observed, and retinotopic adaptation did not transfer to world-centered coordinates after a saccade (Experiment 4). Finally, we found that aftereffects in different reference frames have a similar, narrow spatial tuning profile (Experiment 5). Together, our results suggest that the neural representation of local separation resides early in the visual cortex, but it can also be modulated by activity in higher visual areas.

A PCA-Based Active Appearance Model for Characterising Modes of Spatiotemporal Variation in Dynamic Facial Behaviours.

Faces carry key personal information about individuals, including cues to their identity, social traits, and emotional state. Much research to date has employed static images of faces taken under tightly controlled conditions yet faces in the real world are dynamic and experienced under ambient conditions. A common approach to studying key dimensions of facial variation is the use of facial caricatures. However, such techniques have again typically relied on static images, and the few examples of dynamic caricatures have relied on animating graphical head models. Here, we present a principal component analysis (PCA)-based active appearance model for capturing patterns of spatiotemporal variation in videos of natural dynamic facial behaviours. We demonstrate how this technique can be applied to generate dynamic anti-caricatures of biological motion patterns in facial behaviours. This technique could be extended to caricaturing other facial dimensions, or to more general analyses of spatiotemporal variations in dynamic faces.

Exploring the Common Mechanisms of Motion-Based Visual Prediction.

Human vision supports prediction for moving stimuli. Here we take an individual differences approach to investigate whether there could be a common processing rate for motion-based visual prediction across diverse motion phenomena. Motion Induced Spatial Conflict (MISC) refers to an incongruity arising from two edges of a combined stimulus, moving rigidly, but with different apparent speeds. This discrepancy induces an illusory jitter that has been attributed to conflict within a motion prediction mechanism. Its apparent frequency has been shown to correlate with the frequency of alpha oscillations in the brain. We asked what other psychophysical measures might correlate positively with MISC frequency. We measured the correlation between MISC jitter frequency and another three measures that might be linked to motion-based spatial prediction. We demonstrate that the illusory jitter frequency in MISC correlates significantly with the accrual rate of the Motion Induced Position Shift (MIPS) effect - the well-established observation that a carrier movement in a static envelope of a Gabor target leads to an apparent position shift of the envelope in the direction of motion. We did not observe significant correlations with the other two measures - the Adaptation Induced Spatial Shift accrual rate (AISS) and the Smooth Motion Threshold (SMT). These results suggest a shared perceptual rate between MISC and MIPS, implying a common periodic mechanism for motion-based visual prediction.

Occipital alpha-band brain waves when the eyes are closed are shaped by ongoing visual processes.

One of the seminal findings of cognitive neuroscience is that the power of occipital alpha-band (~ 10 Hz) brain waves is increased when peoples' eyes are closed, rather than open. This has encouraged the view that alpha oscillations are a default dynamic, to which the visual brain returns in the absence of input. Accordingly, we might be unable to increase the power of alpha oscillations when the eyes are closed, above the level that would normally ensue when people close their eyes. Here we report counter evidence. We used electroencephalography (EEG) to record brain activity when people had their eyes open and closed, both before and after they had adapted to radial motion. The increase in alpha power when people closed their eyes was increased by prior adaptation to a broad range of radial motion speeds. This effect was greatest for 10 Hz motion, but robust for other frequencies (and especially 7.5 Hz). This discredits a persistent entrainment of activity at the adaptation frequency as an explanation for our findings. Our data show that the power of occipital alpha-band brain waves can be increased by motion sensitive visual processes that persist when the eyes are closed. Consequently, we suggest that the power of these brain waves is, at least in part, an index of the degree to which visual brain activity is being subjected to inhibition. This is increased when people close their eyes, but can be even further increased by pre-adaptation to radial motion.

fMRI evidence that hyper-caricatured faces activate object-selective cortex.

Many brain imaging studies have looked at the cortical responses to object categories and faces. A popular way to manipulate face stimuli is by using a "face space," a high dimensional representation of individual face images, with the average face located at the origin. However, how the brain responds to faces that deviate substantially from average has not been much explored. Increasing the distance from the average (leading to increased caricaturing) could increase neural responses in face-selective regions, an idea supported by results from non-human primates. Here, we used a face space based on principal component analysis (PCA) to generate faces ranging from average to heavily caricatured. Using functional magnetic resonance imaging (fMRI), we first independently defined face-, object- and scene-selective areas with a localiser scan and then measured responses to parametrically caricatured faces. We also included conditions in which the images of faces were inverted. Interestingly in the right fusiform face area (FFA), we found that the patterns of fMRI response were more consistent as caricaturing increased. However, we found no consistent effect of either caricature level or facial inversion on the average fMRI response in the FFA or face-selective regions more broadly. In contrast, object-selective regions showed an increase in both the consistency of response pattern and the average fMRI response with increasing caricature level. This shows that caricatured faces recruit processing from regions typically defined as object-selective, possibly through enhancing low-level properties that are characteristic of objects.

Visual predictions, neural oscillations and naïve physics.

Prediction is a core function of the human visual system. Contemporary research suggests the brain builds predictive internal models of the world to facilitate interactions with our dynamic environment. Here, we wanted to examine the behavioural and neurological consequences of disrupting a core property of peoples' internal models, using naturalistic stimuli. We had people view videos of basketball and asked them to track the moving ball and predict jump shot outcomes, all while we recorded eye movements and brain activity. To disrupt people's predictive internal models, we inverted footage on half the trials, so dynamics were inconsistent with how movements should be shaped by gravity. When viewing upright videos people were better at predicting shot outcomes, at tracking the ball position, and they had enhanced alpha-band oscillatory activity in occipital brain regions. The advantage for predicting upright shot outcomes scaled with improvements in ball tracking and occipital alpha-band activity. Occipital alpha-band activity has been linked to selective attention and spatially-mapped inhibitions of visual brain activity. We propose that when people have a more accurate predictive model of the environment, they can more easily parse what is relevant, allowing them to better target irrelevant positions for suppression-resulting in both better predictive performance and in neural markers of inhibited information processing.

An observer model of tilt perception, sensitivity and confidence.

Humans experience levels of confidence in perceptual decisions that tend to scale with the precision of their judgements; but not always. Sometimes precision can be held constant while confidence changes-leading researchers to assume precision and confidence are shaped by different types of information (e.g. perceptual and decisional). To assess this, we examined how visual adaptation to oriented inputs changes tilt perception, perceptual sensitivity and confidence. Some adaptors had a greater detrimental impact on measures of confidence than on precision. We could account for this using an observer model, where precision and confidence rely on different magnitudes of sensory information. These data show that differences in perceptual sensitivity and confidence can therefore emerge, not because these factors rely on different types of information, but because they rely on different magnitudes of sensory information.

Recognising the dynamic form of fire.

Encoding and recognising complex natural sequences provides a challenge for human vision. We found that observers could recognise a previously presented segment of a video of a hearth fire when embedded in a longer sequence. Recognition performance declined when the test video was spatially inverted, but not when it was hue reversed or temporally reversed. Sampled motion degraded forwards/reversed playback discrimination, indicating observers were sensitive to the asymmetric pattern of motion of flames. For brief targets, performance increased with target length. More generally, performance depended on the relative lengths of the target and embedding sequence. Increased errors with embedded sequence length were driven by positive responses to non-target sequences (false alarms) rather than omissions. Taken together these observations favour interpreting performance in terms of an incremental decision-making model based on a sequential statistical analysis in which evidence accrues for one of two alternatives. We also suggest that prediction could provide a means of providing and evaluating evidence in a sequential analysis model.

Synchronous facial action binds dynamic facial features.

We asked how dynamic facial features are perceptually grouped. To address this question, we varied the timing of mouth movements relative to eyebrow movements, while measuring the detectability of a small temporal misalignment between a pair of oscillating eyebrows-an eyebrow wave. We found eyebrow wave detection performance was worse for synchronous movements of the eyebrows and mouth. Subsequently, we found this effect was specific to stimuli presented to the right visual field, implicating the involvement of left lateralised visual speech areas. Adaptation has been used as a tool in low-level vision to establish the presence of separable visual channels. Adaptation to moving eyebrows and mouths with various relative timings reduced eyebrow wave detection but only when the adapting mouth and eyebrows moved asynchronously. Inverting the face led to a greater reduction in detection after adaptation particularly for asynchronous facial motion at test. We conclude that synchronous motion binds dynamic facial features whereas asynchronous motion releases them, allowing adaptation to impair eyebrow wave detection.

The interrelationship between the face and vocal tract configuration during audiovisual speech.

It is well established that speech perception is improved when we are able to see the speaker talking along with hearing their voice, especially when the speech is noisy. While we have a good understanding of where speech integration occurs in the brain, it is unclear how visual and auditory cues are combined to improve speech perception. One suggestion is that integration can occur as both visual and auditory cues arise from a common generator: the vocal tract. Here, we investigate whether facial and vocal tract movements are linked during speech production by comparing videos of the face and fast magnetic resonance (MR) image sequences of the vocal tract. The joint variation in the face and vocal tract was extracted using an application of principal components analysis (PCA), and we demonstrate that MR image sequences can be reconstructed with high fidelity using only the facial video and PCA. Reconstruction fidelity was significantly higher when images from the two sequences corresponded in time, and including implicit temporal information by combining contiguous frames also led to a significant increase in fidelity. A "Bubbles" technique was used to identify which areas of the face were important for recovering information about the vocal tract, and vice versa, on a frame-by-frame basis. Our data reveal that there is sufficient information in the face to recover vocal tract shape during speech. In addition, the facial and vocal tract regions that are important for reconstruction are those that are used to generate the acoustic speech signal.

A data-driven characterisation of natural facial expressions when giving good and bad news.

Facial expressions carry key information about an individual's emotional state. Research into the perception of facial emotions typically employs static images of a small number of artificially posed expressions taken under tightly controlled experimental conditions. However, such approaches risk missing potentially important facial signals and within-person variability in expressions. The extent to which patterns of emotional variance in such images resemble more natural ambient facial expressions remains unclear. Here we advance a novel protocol for eliciting natural expressions from dynamic faces, using a dimension of emotional valence as a test case. Subjects were video recorded while delivering either positive or negative news to camera, but were not instructed to deliberately or artificially pose any specific expressions or actions. A PCA-based active appearance model was used to capture the key dimensions of facial variance across frames. Linear discriminant analysis distinguished facial change determined by the emotional valence of the message, and this also generalised across subjects. By sampling along the discriminant dimension, and back-projecting into the image space, we extracted a behaviourally interpretable dimension of emotional valence. This dimension highlighted changes commonly represented in traditional face stimuli such as variation in the internal features of the face, but also key postural changes that would typically be controlled away such as a dipping versus raising of the head posture from negative to positive valences. These results highlight the importance of natural patterns of facial behaviour in emotional expressions, and demonstrate the efficacy of using data-driven approaches to study the representation of these cues by the perceptual system. The protocol and model described here could be readily extended to other emotional and non-emotional dimensions of facial variance.

Motion integration is anisotropic during smooth pursuit eye movements.

Smooth pursuit eye movements (pursuit) are used to minimize the retinal motion of moving objects. During pursuit, the pattern of motion on the retina carries not only information about the object movement but also reafferent information about the eye movement itself. The latter arises from the retinal flow of the stationary world in the direction opposite to the eye movement. To extract the global direction of motion of the tracked object and stationary world, the visual system needs to integrate ambiguous local motion measurements (i.e., the aperture problem). Unlike the tracked object, the stationary world's global motion is entirely determined by the eye movement and thus can be approximately derived from motor commands sent to the eye (i.e., from an efference copy). Because retinal motion opposite to the eye movement is dominant during pursuit, different motion integration mechanisms might be used for retinal motion in the same direction and opposite to pursuit. To investigate motion integration during pursuit, we tested direction discrimination of a brief change in global object motion. The global motion stimulus was a circular array of small static apertures within which one-dimensional gratings moved. We found increased coherence thresholds and a qualitatively different reflexive ocular tracking for global motion opposite to pursuit. Both effects suggest reduced sampling of motion opposite to pursuit, which results in an impaired ability to extract coherence in motion signals in the reafferent direction. We suggest that anisotropic motion integration is an adaptation to asymmetric retinal motion patterns experienced during pursuit eye movements. NEW & NOTEWORTHY This study provides a new understanding of how the visual system achieves coherent perception of an object's motion while the eyes themselves are moving. The visual system integrates local motion measurements to create a coherent percept of object motion. An analysis of perceptual judgments and reflexive eye movements to a brief change in an object's global motion confirms that the visual and oculomotor systems pick fewer samples to extract global motion opposite to the eye movement.

Suboptimal human multisensory cue combination.

Information from different sensory modalities can interact, shaping what we think we have seen, heard, or otherwise perceived. Such interactions can enhance the precision of perceptual decisions, relative to those based on information from a single sensory modality. Several computational processes could account for such improvements. Slight improvements could arise if decisions are based on multiple independent sensory estimates, as opposed to just one. Still greater improvements could arise if initially independent estimates are summed to form a single integrated code. This hypothetical process has often been described as optimal when it results in bimodal performance consistent with a summation of unimodal estimates weighted in proportion to the precision of each initially independent sensory code. Here we examine cross-modal cue combination for audio-visual temporal rate and spatial location cues. While suggestive of a cross-modal encoding advantage, the degree of facilitation falls short of that predicted by a precision weighted summation process. These data accord with other published observations, and suggest that precision weighted combination is not a general property of human cross-modal perception.

Personality Traits Do Not Predict How We Look at Faces.

While personality has typically been considered to influence gaze behaviour, literature relating to the topic is mixed. Previously, we found no evidence of self-reported personality traits on preferred gaze duration between a participant and a person looking at them via a video. In this study, 77 of the original participants answered an in-depth follow-up survey containing a more comprehensive assessment of personality traits (Big Five Inventory) than was initially used, to check whether earlier findings were caused by the personality measure being too coarse. In addition to preferred mutual gaze duration, we also examined two other factors linked to personality traits: number of blinks and total fixation duration in the eye region of observed faces. No significant correlations were found between any of these measures and participant personality traits. We suggest that effects previously reported in the literature may stem from contextual differences or modulation of arousal.

A spatial frequency spectral peakedness model predicts discrimination performance of regularity in dot patterns.

Subjective assessments of spatial regularity are common in everyday life and also in science, for example in developmental biology. It has recently been shown that regularity is an adaptable visual dimension. It was proposed that regularity is coded via the peakedness of the distribution of neural responses across receptive field size. Here, we test this proposal for jittered square lattices of dots. We examine whether discriminability correlates with a simple peakedness measure across different presentation conditions (dot number, size, and average spacing). Using a filter-rectify-filter model, we determined responses across scale. Consistently, two peaks are present: a lower frequency peak corresponding to the dot spacing of the regular pattern and a higher frequency peak corresponding to the pattern element (dot). We define the "peakedness" of a particular presentation condition as the relative heights of these two peaks for a perfectly regular pattern constructed using the corresponding dot size, number and spacing. We conducted two psychophysical experiments in which observers judged relative regularity in a 2-alternative forced-choice task. In the first experiment we used a single reference pattern of intermediate regularity and, in the second, Thurstonian scaling of patterns covering the entire range of regularity. In both experiments discriminability was highly correlated with peakedness for a wide range of presentation conditions. This supports the hypothesis that regularity is coded via peakedness of the distribution of responses across scale.

Selective binding of facial features reveals dynamic expression fragments.

The temporal correspondence between two arbitrarily chosen pairs of alternating features can generally be reported for rates up to 3-4 Hz. This limit is however surpassed for specialised visual mechanisms that encode conjunctions of features. Here we show that this 3-4 Hz limit is exceeded for eye gaze and eyebrow pairing, but not for eye gaze and mouth pairing, suggesting combined eye and eyebrow motion constitutes a dynamic expression fragment; a building block of superordinate facial actions.